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ABSTBACT 

This study described the initial developnent of a 
Connittee Bole Bating Scale (CBBS) and a Conference Bole Semantic 
Differential (CBSD) as neasures of these specific roles connon to 
adnlnistrators at all educational levels. Although the CBBS shoved 
general construct validity, the snail amount of common variance 
accounted for by the factors and the low internal consistency 
reliability of the initial form of CBBS Indicate a need for further 
development before nonresearch use. Because it is further developed 
than the CBBS, the CBSD showed greater construct validity, and a 
factor structure which accounts for more common variance. In 
addition, the CBSD has an acceptable internal consistency reliability 
of .86 and the general stability of its factors over time has been 
demonstrate?. However, the internal consistency reliability of the 
factors should be increased by adding related scales to permit 
diagnostic use of factor scores. Scale substitutions or modifications 
should be made, also, to further increase the independence of the 
factors before this instrument is used widely. (Author/SM) 
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A growing concern for administrator evaluation has accompanied the educa- 
tional accountability movement. On the basis of a review of the ERIC litera- 
ture on evaluation, Poliakoff (1973) reported that "by 1968 a growing trend to 
evaluate school administrators was evident." 

Although writers such as Barraclough (1973), McCleary (1973) and Melton 
(1970) cite the need for development of a variety of measures of increasingly 
complex administrative roles at all levels of education, according to Barraclough 
(1973) and Scriven (1974), procedures for evaluating administrative performance 
are underdeveloped. 

In this pilot study, a Committee Role Rating Scale (CRRS) and a Conference 
Role Semantic Differential (CRSD) were developed and field tested as brief 
diagnostic measures of these educational leadership roles. 



The need for valid and reliable, diagnostic measures of administrator per- 
formance was cited by Castetter and Heisler (1971), Rosenberg (1971), and by 
Wochner and Lynch (1973). 

In contrast to the multitude of studies on technical characteristics of 
faculty performance measures, studies of administrative evaluation lack such 
characteristics. Instead, they focus on identification of personnel evaluated, 
frequency of evaluation, evaluation forms or instruments, notification of 
results and appeal procedures. Stemnock, for example, has reported four 
studies of administrative performance for the Educational Research Service. 
Her 1968 study included the formalized evaluation procedures of 62 school systems. 
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Her 1970 report presented the **cUent-oriented'' evaluation procedures of 29 
school systems. In 1971, Stemnock's report included evaluation forms of 11 
school systems. These forms were intended to stimulate the thinking of indi- 
viduals Involved in developing procedures for evaluating administrators. In 
her 1973 report, she summarizes the client-centered-evaluation procedures of 
469 school systems. This report Includes 10 administrator evaluation forms. 
In 1973, Napa College described Us procedures for evaluating college admin- 
istrators. However, none of these studies included data on the technical 
characteristics of any of the instruments presented. 

Because of its ability to measure complex concepts by means of a simple 
format, the semantic differential has been used widely in educational research. 
In studies of teacher effectiveness, the semantic differential has been shown 
to possess desirable technical characteristics. For example, Gulo (1972) 
compared the results of four studies in which the semantic differential was 
used to evaluate the teaching effectiveness of college professors. He concluded: 

• some teacher effectiveness factors are stable across time and 
populations 

• the proportion of variance accounted for by each factor varies 
across populations and from one factor analysis to another 

• the semantic differential seems to be an especially useful 
technique for quantifying cmergunt variables afssociated with 
student perceptions of teaching effectiveness and effective 
professors. 

In addition to its use in faculty evaluation, the semantic differential has 
been used in administrator evaluation. 

Although no data on its technical characteristics are given, Stemnock*s 
1970 report Included a semantic differential which has been used to evaluate 
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principals In Cheyenne, Wyoming. This instrument, which consisted of 18 adjec- 
tive and verb-phrase pairs, provides a global assessment of the principalis role. 

Procedures 

A Committee Role Rating Scale (CRRS) (Figure 1) was designed to sample task, 
group maintenance and self-serving functions in committee meetings. On the 
basis of these functional dimensions of leadership described by Lassey (1971, 
pp. 5- four statements were developed which represented behaviors character- 
istic of each function. To simplify diagnostic scoring, items were listed by 
function. A five -choice response scale ranging '^always" to "never" was employ- 
ed to record frequency of occurrence of each behavior. 

In addition to the 12 objective items, the CRRS Included two free-response 
items whinh asked evaluators to idonLlfy strengths and weaknesses in the 
committee functions performed by the person being evaluated. 



Insert Figure 1 about here 



CRRS was completed anonymously by each member of each committee on which 
three institutional research staff members served. Responses from all 23 
respondents were pooled in a factor analysis to assess the construct validity 
of the three hypothesized factors. 

On the basis of the factor analysis, factor scores were calculated by 
summing across the weighted responses for the statements comprising each factor. 
For each factor and for the entire scale, internal consistency reliability was 
determined according to Cronbach^s Alpha. 

Prompted by the semantic differential used for teacher t^val^iation of 
principals in Cheyenne, Wyoming (Stemnock, 1970, p. 42), a Con ference Role 
S eman t i c D 1 f f e ren t i al (CRSD) (Figure 2) was designed to sample task and 
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Interpersonal functions of educational leaders in individual conferences • 
Seven bi-polar adjective pairs were selected to represent each function. Four 
adjective-scales were drawn from the Cheyenne instrument. The remainder were 
drawn or adapted from Osgood (1957, pp. 53-61) using his criteria for scale 
selection (pp. 78-80). Item order and adjective polarity were randomized to 
minimize response bias. A seven interval response scale was used. 



Insert Figure 2 about here 



In addition to the 14 adjective-scales, the CRSD included two free-response 
items which asked evaluators to identify strengths and weaknesses in the con- 
ference roles performed by the person being evaluated. An unsigned draft CRSD 
was completed by 52 staff members. In the judgment of three institutional 
research staff members each evaluator had sufficient conference experience 
to quality him for the task. As with CRRS, responses from all 52 individuals 
Were pooled for factor analysis of construct validity. 

Factor scores and their internal consistency reliability were determined 
in the same manner as CRRS. In addition, factor stability was determined by 
applying Veldman*s (1967, pp. 242-244) "relate procedure" to the 1972 and 
1973 CRSD factor loadings. 

All statistical calculations were performed by an IBM 370 computer using 
programs TESTAT, FACTOR and RELATE (Veldman, 1967, pp. 174-180, 222-236 and 
242-244). 

A revised CRSD (Figure 3) was administered one year after the draft scale. 



Insert Figure 3 about here 
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ResultB 

As shorn In Table 1, for the CRRS, principal components analysis with 
varimax rotation to simple structure yielded Group Maintenance, Self-Serving 
and Task Factors which accounted for 50 per cent .of the total common variance* 



Insert Table 1 about here 



However, the items didn't cluster exactly as hypothesized in scale con- 
struction. Two intended ^'task*' items joined the "group maintenance" factor 
and three iLems appeared on no factor because there was no variance in re- 
sponses. Further testing on a larger sample should establish whether lack 
of variance is attributable to the nature of these scale items or to the 
performance of those being evaluated. 

As shown in the following table, internal consistency reliability co- 
efficients for the CRi<S ranged from .45 to .75 with the factor consisting 
of the largest number of items having the largest alpha. 

Internal Consistency Reliability of CRRS 



Number Coefficient 
Factor of Items Alpha 

Group Maintenance 5 .75 

Self-Serving 2 .54 

Task 2 .45 

CRRS 9 .57 



N « 23 

As shown in Table 2, for the draft CRSD, principal components analysis 
with varimax rotation to simple structure yielded four factors— one task, 
two Interpersonal and one task and interpersonal. 
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In an effort to obtain clearer factors, CRSD was revised by omitting an 
adjective-scale which loaded almost equally on two factors and by adding 
three adjective-scales intended to help define three factors. As shown in 
Table 3 principal components analysis of varlmax rotation to simple structure 
for the revised CRSD yielded one task and three interpersonal factors which 
accounted for 70 per cent of the variance. The loadings suggested these 
factor descriptions: Problem-Solving, Tact in Personal Contacts, Accepting 
Others' Views and Persuasive. These factors are similar to several estab- 
lished by Brown (1967) in his study ot job analysis by multidimensional 
scaling. 



Insert Table 3 about here 



Although these factors were clearer than those obtained with the draft 
CRSD, shifts in factor location for some scales indicate a need for further 
validation on a larger sample. 

Internal consistency reliability of CRSD factors ranged from .39 to .91 

with the factor comprised of the i/irgost number of Items having the largest 

coefficient. The revised CRSi) instrument had an alpha of .86 as shown in 

the following table. 

Internal Consistency Reliability of CRSD 





Number 


Coefficient 


Factor 


of Items 


Alpha 


Tact in Personal Contacts 


7 


.91 


Accepting Others' Views 




.64 


Persuasive 


2 


.63 


Problem-Solving 




.39 


CRSD 


15 


.86 


N « 52 







Factor stability from 1972 to 1973 was established by applying Veldraan's 
(1967) "relate procedure" to maximize the fit between 1972 and 1973 factor 
solutions. The cosines of the angles between the factors produced In the 
"relate procedure" are shown in the following table. These cosines can be 
interpreted in the same way as correlation coefficients between factors. 





Stability 


of CRSD 


Factors 




1972 Factors 




1973 


Factors 






1 


2 


3 


4 


1 


.868 


-.221 


-.438 


.149 


2 


.118 


-.299 


.625 


.712 


3 


.496 


.534 


.579 


-.366 


4 


-.053 


.760 


-.288 


.580 



Factor 1 in the 1972 solution is substantially correlated with Factor 1 
in the 1973 solution, and these factors are comparatively uncorrelated with 
the other three factors. 

Factors 2 and 4 in these solutions are interchangeable. 

Although the third factor in the 1972 solution is related to the third 
factor in the 1973 solution, the third factor of the 1972 solution is also 
related to the second factor and somewhat to the first factor of the 1973 
solution. 

These results Indicate a substantial stability of CRSD factors from 1972 
to 1973. 

The adjective-scale vector correlations shown in Table 4 indicate that 
the differences in ''meaning" of the factors derived in the two analyses are 
attributable largely to the adjective-scales ''Approachable-Unapproachable 
"Accurate-Careless" and "Pleasant-Annoying" all of which changed factor 
locations from one solution to the other* 



Insert Table 4 about here 



BEST COPY AVAIIABU 

r 8 

Summary 

This pilot study has described the initial development of a Committee Role 
Ra ting Scal e (CRRS) and a Conference Role Semantic Differential (CRSD) as 
measures of these specific roles common to administrators at all educational 
levels* 

Although the CRRS showed general construct validity, the small amount of 
common variance accounted for by the factors and the low internal consistency 
reliability of the initial form of CRRS indicate a need for further development 
before non-research use. 

Because it is further developed than the CRRS, the CRSD shows greater con- 
struct validity, and a factor structure which accounts for more ccmmon 
variance. In addition, the CRSD has an acceptable internal consistency reli- 
ability of .86 and the general stability of its factors over time has been 
demonstrated. However, the internal consistency reliability of the factors 
should be increased by adding related scales to permit diagnostic use of 
factor scores. Scale substitutions or modifications should be made, also, 
to further increase th*! independence of the factors before this instrument 
is used widely. 

Other limitations of these instruments also reflect their developmental 
stage. These include: 

• Use of only three community collegt^ Institutional researchers as 
the sample evaluated. 

• Use of only 23-52 community college staff members as evaluators. 

• Lack of norms for interpretation of relative performance. 

In spite of these presei:': limitations, refined versions of these instru- 
ments should be helpful in staff development, contract renewal and salary 
determination. Because of the universality of the specific educational roles 
assessed by these instruments, they should be applicable to all levels of 
^ administrators at all levels of education. 

ERIC 



9 



References 

Barraclough, Terry. Adn anlstrator Evaluation , Eugene, Oregon? ERIC Clearing- 
house on Educational Management » 1973. 

Brown, Kenneth R. "Job Analysis by Multidimensional Scaling," Journal of 
Applied Psychology , Vol. 51, No. 6, pp. 469-475, 1967. 

Castetter, William B. and Heister, Richard S. Appraising and Improving the 

Performance of School Administrative Personnel . Philadelphia: Graduate 
School of Education, University of Pennsylvania, 1971. (ED 060 540) 

Gulo, E. Vaughn. Measuring D imensions of Teaching Effectiveness with the 
Semantic Differential , 1972. (ED 064 346) 

Lassey, William R. "Dimensions of Leadership," Leadership and Social Chang e, 
by University Association, Associates Publishers and Consultants, 
Iowa City, Iowa, 1971. 

McCleary, Lloyd E. Competency Based Educational Administration and Applications 
to Related Fields , January, 1973. (ED 077 136) " 

Melton, George E. and others. The Prlnclpalshlp; Job Specifications and Salarv 
Considerations for the '70's . Washington, D.C.: National Association of 
Secondary School Principals, 1970. (ED 040 492) 

Napa College. Evaluation of Administrators and Practices for Evaluating Admin- 
istrators . Napa, California} Napa College, 1973. (ED 081 414) 

Osgood, Charles E.; Sucl, George J. and Tannenbaum, Percy H. The Measurement 
of Moanlna , University of Illinois Press, 1957. 

Poliakoff, Lorraine. Evaluating School Personnel Today , Washington, D.C. : 
ERIC Clearinghouse on Teacher Education, 1973. (ED 073 045) 

Rosenberg, Max. "The Values of School Principal Evaluation," Education . 
91, 3, pp. 212-214, 1971. 

Scriven, Michael. Oral presentation Included In Evaluation Pre-session of the 
American Educational Research Association, Chicago, Illinois, April 14- 
15, 1974. 

Stemnock, Suzanne K. Evaluating Administrative Performance . Educational 
Research Service Circular Number 7, American Association of School 
Administrators: Washington, D.C, 1968. (ED 032 635) 

Stemnock, Suzanne K. The Eyal uatee Evaluates the Evaluator . Educational 

Research Service, Circular Number 5, American Association of School 
Administrators: Washington, D.C, 1970. (ED 044 378) 

Stemnock, Suzanne K. Ev aluating Administrative/Supervisory Performance . 

Educational Research Service, Circular Number 6, American Association 
of School Administrators: JVashington, D.C, 1971. (ED 058 155) 



ERIC 



BESt COPY AVAILABLE 



10 

Stemnock, Suzanne K. Evaluatin g the Evaluator . Washington, D.C.; Educational 
Research Service, 1973. (ED 081 114) 

Veldman, Donald J. Fortran Programming for the Behavioral S ciences. New York? 
Holt, Rlnehart and Winston, 1967. 

Wochner, Raymond E. and Lynch, Steven B. The Identification and Development 
of Administrative Competencies . Tempo, Arizona: Arizona State 
University, 1973. 



FIGURE 1, Committee Role Rating Scale 
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Committee Member Evaluation of 



Activities on 1972-73 ^[ committee. 

1. Before each activity, circle the letter which represents yrar rating of 
this committee member. Use this key: 

^ A = Always 

B = Usually 

^ C = Sometimes 

^ D = Seldom 

E = Never *' 
Blank = Not Observed 



A 


B 


C 


D 


E 


1. 


A 


B 


C 


D 


E 


2. 


A 


B 


C 


D 


E 


3. 


A 


B 


c 


D 


E 


4. 


A 


B 


c 


D 


E 


5. 


A 


B 


c 


D 


E 


6. 


A 


B 


c 


D 


E 


7. 


A 


B 


c 


D 


E 


8. 


A 


B 


c 


D 


E 


9. 


A 


B 


c 


D 


E 


10. 


A 


B 


c 


D 


E 


11. 



committee decisions. 

4. Pulled together related ideas discussed by the group. 

Promoted good personal feelings among committee 
members . 

Encouraged others to contribute to the discussion. 

Listened well to contributions of others. 

Was courteous to and respectful of others. 

Criticized or blamed others; deflated the ego or 
status of others. 

Attempted to call attention to himself by loud or 
' excessive talking. 



Interfered with group progress by going off on a 
tangent, arguing too much on a point or rejecting 
ideas without consideration. 

A B C D E 12. Introduced or supported suggestions related to his 

own pet concerns or philosophies. 

II. What contributes most to this staff members' effectiveness as a committee 
member? 

III. In what ways could this staff member increase his effectiveness as a 
committee member? 
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FIGURE 2. Conference Role Semantic Differential (Draft) 
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STAFF EVALUATION OF 



PART !♦ Directions: Place an "X" on each line below, at the point which 

represents your judgment of this staff member. 



unapproachable 



annoying 
approving 
attentive I 



closed-minded |_ 



constructive ideas |_ 
impulsive l 



accurate L 
gets to point |_ 

superficial | 



prompt |_ 
ignores problems | 



No 



Very Quite Slightly Opinion Slightly Quite Very 



L 



J approachable 



J pleasant 

J faultfinding 
Inattentive 
open-minded 
worthless Ide 
deliberate 



J careless 



J roundabout 
J analytical 
I slow 



J solves proble 



PART II. Directions: Answer in the spaces below. You may want to describe 

an Incident to illustrate your comments. 



1. WHAT CONTRIBUTES MOST TO THIS STAFF MEMBER'S JOB EFFECTIVENESS? 



2. IN WHAT WAYS COULD THIS STAFF MEMBER INCREASE HIS JOB EFFECTIVENESS? 
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FIGURE 3. Conference Role Semantic Differential (Revised) 



STAFF EVALUATION OF 



PART I. Directions: Place an "X" on each line below » at the point which 

represents your judgment of this staff member* 



Very, 



accepting 
unapproai'hable 
annoy ::.ng 
approving 
attentive 
closed-minded 
constructive ideas 
impulsive 
accurate 
gets to point 
superficial 



prompt 
Ignores problems [ 
dictates [ 
clarifies 



No 

Quite Slightly Opinion Slightly 



1 



rejecting 
approachable 
pleasant 
faultfinding 
inattentive 
open-minded 
worthless idea 
deliberate 
careless 
roundabout 
analytical 
slow 

solves problen 
persuades 
confuses 



PART II. Directions J Answer in the spaces below. You may want to describe 

an incident to illustrate your cotmnentSi 



1. WHAT CONTRIBUTES MOST TO THIS STAFF MEMBER^R JOB EFFECTIVENESS? 



2. IN WHAT WAYS COULD THIS STAFF MEMBER INCREASE HIS JOB EFFECTIVENESS? 



TABLE 1. 1973 Rotated Factor Loadings for Committee Roles of 

Institutional Research Personnel 

Factors I« ^ II, III* Coromunalltv 
I. Group Maintenance 

Pulled together related Ideas dla- 

cussed by the group 869 130 -316 872 
Listened well to contributions of 

others 846 -142 -195 774 
Promoted good personal feelings 

emn^ committee members 678 -243 148 540 
Encourpsed others to contribute to 

the dlocusslon 640 188 399 604 

Contributions were clearly stated .511 114 432 460 

H, Self-Servin»5 

Interfered with group progress by 
going off on a tangent} arguing 

too much on a point or rejecting 

ideaa without conolderation 20 875 -144 786 

Introduced or supported sufl;gestions 
related to his own pet concerns or 

philosophies - 9 781 -257 683 

III* Task 

Introduced information or ideas which 

contributed to committee decisions 35 229 879 826 

Was well prepared for committee meet- 
ings - 58 - 69 593 360 

Per Cent of Total Variance 22 14 14 

^ Leading decimals have been omitted from all factor loadings 
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TABLE 2. 1972 Rotated Factor Loadings fo 

Research 



Factors 

I. Person-Task 

Deliberate— Impulsive 
Analytical— Superficial 
Approachable-- Unapproachable 
Attentive— Inattentive 
Cooperative— Uncooperative 



II. P r oh lem- S 0 1 vln g 

Accurate— Careless 

Solves Problems— Ignores Problems 

Constructive Ideas — VJorthless Ideas 



III. Tact in Personal Contacts 

Gets to Point— Roundabout 
Pleasant— Annoying 
Prompt— Slow 

Cooperative— Uncooperative 



IV. Accepting Others' Views 

Approving— Faultfinding 
Open-Mlnded— Closed-Minded 



Per Cent of Total Variance 



d Leading decimals have been omitted 



' Conference Roles of Institutional 
Personnel 



I V ^ II* Illy IV. Communalitv 



866 


- 31 


72 


315 


855 


853 


310 


89 


29 


832 


741 


230 


435 




f oU 


701 


161 


79 


•< 45 




536 


244 






AAA 
ooo 


31 


885 


-113 


225 


847 


318 


865 


175 


72 




289 


825 


287 


78 


853 


159 


42 


895 


- 84 


835 


• 23 


81 


714 


333 


629 


378 


67 


664 


339 


704 


536 


244 


545 


28 


688 


28 


37 


172 


923 


884 


119 


402 


27 


771 


' 771 


40 


16 


13 


9 





from all factor loadings 
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TABLE 3. 1973 Rotated Factor Loadings for Conference Roles of Institutional 

Research Personnel 



Factors 


I. a 


II. 




IV. 


Communal ity 


Tact in Personal Contacts 








- 




Gets to Point— Roundabout 


908 


90 


8 


140 


852 


1 m G »M M ^m. ••■AM ^» <<« 

LlSLTClt 108'"""Lon£USeS 


Q C /. 


15o 


73 


f\ OA 

280 


838 


AunenLive'"""inaCuentive 


OR 1 


16 


00 


129 


•fin 

748 


ri.eas£inL-*"*Annoying 






201 


116 


878 


Analytical— Superficial 


778 


50 


237 


224 


714 


Pronpt*- Slow 


530 


257 


345 


366 


600 


Ac cur a t e— • Car e 1 es s 


104 


18 


427 


343 


555 


Accepting Others' Views 












Accepting— Re j ecting 


30 


727 


17 


19 


530 


Approachable— -Unapproachable 


438 


615 


94 


87 


585 


Approving—Faultfinding 


80 


584 


90 


514 


619 


Open-Minded--CloBed-Minded 


351 


562 


211 


530 


764 


Persuasive 












Persuades — Dictates 


36 


361 


830 


16 


821 


De liber ate— Impulsive 


443 


137 


740 


4 


762 


Problem-Solving 












Constructive Ideas— Worthless Ideas 


100 


116 


163 


798 


686 


Solves Problems— Ignores Problems 


291 


117 


238 


606 


522 


Per cent of Total Variance 


32 


13 


12 


13 




^ Leading decimals have been omitted from all factor loadings 
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TABLE A. Adjective Scale-Vector Correlations and Factor Locations 



Factor Location 



Adjective Scale 


r 


1972 




1973 






1 




0 
4 


Accurate-^'-Ca^re.less 


.696 


2 




1 


Pleas an t— Annoy 1 ng 


.713 


3 




1 


Conotructlve Ideas — Worthless Ideas 


.781 


2 


a 


4 


C^'^.tr, to Pcf nt^'-Roundabout 


.793 


3 




1 


P 3 l-\bor:at:n— Impulsive 


.873 






3 




.880 






3 


Approving— Faultfinding 


.908 




a 


2 


At tnntivo— Inattentive 


.916 




a 


1 


Opon-Ilin'5.od— 'Close-Minded 


.922 




■ 


2 


Ann^7 1 i c at — Pupo rf i ci al 


.9A1 




a 


1 


Solves Prchlei^s— Ignores Problems 


.986 


2 


a 


4 



